yelbatinfandomcom-20200214-history
Automated Rating System
__TOC__ Ratemall Corporation uses the Automated Ratings System (ARS) for promotion, award and assignment recommendations, as well as for counseling and performance improvement. The ARS is not a typical 360º evaluation. Every 360º system requires an internal mechanism for assessing the credibility of responses. Typically, this involves lengthy rating survey forms in which dozens of questions are repeated, re-phrased, and couched in various contexts. This allows the system to look for inconsistencies, which indicate that the rater is not carefully considering the answers being given. Unfortunately, it also means that the system is burdensome and impractical for universal application. If I am to be rated, for example, by many people, in order to give a very reliable appraisal of my performance, then I, myself, will have to rate many people. A significant part of my work day will be spent on ratings rather than on the job. The ARS takes an entirely different approach! ARS rating forms are simple, short, quick, and easy, normally completed in less than 2 minutes. This allows everyone to rate many people over the course of a few months without any noticeable impact on their primary jobs. The ARS validates the responses statistically. Rather than looking for different responses to the same re-phrased question, the ARS looks for responses to one question that are at odds with the consensus of opinion gathered from many people on that same question. This is used both to ensure that overall appraisals are statistically reliable and to "rate the rater" in a way that continually improves the quality of ratings. The ARS is a "system of fives." Every employee is required to select one of five responses to five factors evaluating five other employees every month. The persons to be rated each month are selected by the ARS from the employee's "Core List," which contains the names of at least five persons whom the employee must regularly rate. The employee's "Extended List" contains the names of as many as 15 other persons whom the employee must occasionally rate. Home Page When visiting the ARS Web site, an employee sees his/her own Home Page, but can navigate from there to the page for any other employee. Let's assume that you are an employee named Jonathan Studeman. You have already rated three employees this month. Your supervisor is Roger Galloway. Roger's own supervisor is referred to as your manager. With a mouse click on your ARS page, you have just selected the name of another employee, Jane Doe, which is now highlighted. Your page looks like this. Core List and Extended List Each month the ARS selects the names to be rated that month from a list of persons that the employee is required to rate, called the "Core List," and also from a list of persons that the employee is merely eligible to rate, called the "Extended List." The Core List always contains at least five names, and the two lists, combined, will contain at least 20 names. To be eligible for promotion or award, an employee must rate five employees each calendar month. If, at the end of the month, an employee has not done so, the ARS will require that more be rated the next month, and the employee regains eligibility when the deficit is made up. The supervisor and the employee must agree on a list of no fewer than five names for the Core List. These names can be changed at any time, so long as the supervisor and the employee agree, but must at all times include the supervisor, the supervisor's other direct subordinates (those that have regular contact with the employee), and the employee's own immediate subordinates if the employee is a supervisor. If that number is fewer than five, the supervisor and employee will need to select others with whom the employee has regular professional contact in order to make up the difference. (The manager can and should be included if the manager has regular professional contact with the employee.) At any time, the supervisor or the employee can make whatever changes to the Core List seem desirable or necessary, as when someone on the list changes jobs. If the supervisor adds or removes a name, a new proposed list is submitted to the employee for approval, and if the employee adds or removes a name, a new proposed list is submitted to the supervisor for approval. This continues until a new list proposed by one of them is approved by the other without changes. If never changed, the Core List must be re-approved by both at least once per year. The ARS itself generates the Extended List, using names selected from: :* Core Lists of other employees; :* email aliases associated with the employee's office; and, :* organizational charts. The Core List and Extended List are displayed side-by-side. To add a name to the Core List, a name is simply dragged there from the Extended List or from the Global List (all other employees). To remove a name from the Core List, the name is selected and dragged into the Extended List or out into the Global List. Ratings Only five rating factors must be addressed, and there are only five possible ratings for each. In this example, you are rating an employee who is intelligent, highly educated and has taken a lot of corporate training courses. He is the most highly qualified expert that you know of, and in your experience he frequently comes up with new and better ways of doing business. Unfortunately, he is also arrogant, abrasive,and unreliable. Often he is absent or late for work and for meetings. You have frequently observed him missing deadlines, failing to complete his assignments, and making evasive excuses for every lapse, often angrily blaming others. In working groups, everyone looks to him as a leader, acknowledging his expertise and the value of his ideas, but he is usually argumentative, uncooperative, rude, and dismissive of the ideas of others. How would you choose to rate such a person for each of the five factors? Do not click to indicate a repsonse. This is not a working form. Simply point the mouse cursor at the radio button under each option to see what the option means. Reports For every employee, the ARS generates and continually updates a 3-part report that is based on all the ratings submitted by all employees. The ARS report is hidden from everyone until and unless it is "valid," and then it is made available only to authorized viewers. :* To be valid, current ratings by 10-or-more credible others (see below) must reflect a consensus of opinion on all rating factors. :* Current means that the ratings were gathered within the past 6 months. :* Consensus exists when the standard deviation of the ratingsStandard deviation is a statistical measure of how widely or narrowly a range of values spreads from the mean (average) value. A small standard deviation (less than 1) indicates that the values are grouped very tightly around the mean. A large standard deviation indicates that the values are spread out more evenly across the range of possible values. is less than one. ; Pre-authorized Access: Only the employee, the supervisor and the manager have continuous, pre-authorized access to a report. The members of the Promotion and Awards Board also have access to the report, but only when the Board has received a recommendation for promotion or award. The Board loses its access after a determination is made. ; Access Upon Request: An employee may, from time to time, grant 10-day access to other persons, and should normally do so when applying for a new assignment or when recommended for a new assignment. An access request is generated whenever a person who is not pre-authorized to view a report attempts to access the report. The person who attempts such access is asked to either select the purpose from a short list or type the purpose into a text box. The employee is then automatically notified and is required to explicitly grant or deny access. If the request is being made for one of the standard purposes on the list, the employee is required to state a reason if denying the request. Part I: Credibility Assessment The report begins with an assessment of how well or poorly the employee has rated other employees. This is based on a statistical analysis of the ratings submitted by the employee in comparison with the ratings of the same persons by other employees, as well as with the ARS credibility assessments of all employees. If, for a given person and a given factor, the rating submitted by the employee differs from the mean rating by more than one and by more than the standard deviation, that rating is flagged as "questionable." In other words, if a rating is outside the range of the standard deviation but is still only "off" by one (e.g., below average versus average), it is nonetheless accepted as at least "credible." And, if a rating is "off" by more than one but is still within the range of the standard deviation, that rating, too, is still accepted as credible. The only ratings flagged as questionable are those that differ from the average rating by more than one and that lie outside the range of the standard deviation. Individual flags are not revealed in the report. Instead, the system looks at the ratio of "questionable" to "credible" ratings and compares it with the system-wide average. * If the employee has a ratio that is high to a statistically significant degree, the report says, "Less than credible. This employee has submitted a significantly high number of ratings that differ widely from the consensus." * If the employee has a ratio that does not differ from the corporate average in a statistically significant way, the ARS reports that the employee's participation in the system has been "Credible." * If the employee has not submitted the required number of ratings, the system reports that the employee's credibility is "Not assessible." Such an employee is not eligible for promotion or award. Part II: How the Employee was Rated by Others The report next shows an assessment of the employee for each of the five factors, as well as an overall assessment, based on how he was rated by credible others. This is based on the mean rating for each factor for which there was a consensus (standard deviation of less than 1), which excludes any ratings that were not credible (were more than one position off the mean rating and were outside the range of the standard deviation). If there is no consensus for a factor, the entire report is deemed not valid and remains inaccessible.The ARS is designed to avoid and remedy "No consensus" situations by bringing in more names and systematically shifting names in and out of every employee's extended list. This ensures that each employee is rated by a sufficient number of persons who are close enough to the employee to have a credible first-hand opinion on most or all of the rating factors.. Part III: The ARS Recommendation The report concludes by making a recommendation to the Board or to other decision makers. To arrive at this final ARS recommendation, points are awarded for the credibility assessment from Part I and for the overall rating reflected in Part II, and then simple addition produces the result. Four points equates to Strongly Recommended. Three points earns Recommended. Two points are Worthy of Consideration. One means Eligible for Consideration, and zero points equals Ineligible. As shown, every employee who is in good standing within the ARS and has an overall rating of Acceptable is at least eligible to be considered for an award, a promotion or a challenging assignment. * But to be deemed worthy of consideration, an average employee must have participated in the ARS in a credible manner. * Similarly, every superior employee will automatically be recommended unless his/her participation has been less than credible. * And, likewise, every excellent employee will be strongly recommended, assuming that the employee has performed credibly within the system. This provides a very strong and necessary incentive for employees to rate carefully and without bias. Ill-considered ratings harm the employee who submits them by significantly lowering his or her chances of being promoted or awarded. Biased ratings have the exact same effect. And what is more, ill-considered or biased ratings cannot in any way benefit or harm the persons who receive them! For example, if I have a friend who, for a particular factor, is a below average employee and I give this friend a superior rating simply because he is a friend, that rating will do harm to my own credibility and will do him no good. Since it is likely to lie far from the consensus, it will be discounted as not credible. Therefore, my friend and I both know, going in, that I cannot possibly help a friend — nor harm an enemy — by submitting a biased rating. I can only harm myself. Examples A Superior Employee An Excellent Employee An Inferior Employee Remember John? The consensus of opinion tracks very well with how you yourself would have rated him. He has an overall inferior rating, but he would still have been reported as eligible to be considered for award or promotion if he had contributed credible ratings to the ARS system. Unfortunately, he has not done so. He has a tendency to not do his ratings until the last minute, or after a backlog of required ratings has piled up, and then he speeds through them without much thought, and tends to give unreasonably low ratings to persons he just doesn't like, including his supervisor and manager. The result is an assessment of less-then-credible, which pushes him down further, to the lowest level, "Should not be considered." Recommendations Award and Promotion Supervisors recommend, managers endorse, and a corporate Board decides on promotions and awards. Forms for promotion or award recommendation include an option button for the manager's endorsement of the recommendation. If the option "Not endorsed" is selected, the manager must provide a brief explanation. If a manager endorses a recommendation, the Submit button sends the recommendation to the Board. The manger can first add content to the recommendation by using the Explanation text box that is below the Endorse button. If a manager does not endorse the recommendation, he/she must state a reason in the Explanation box, and then the Submit button sends the form back to the supervisor, who must decide on one of three options. # Do nothing. The recommendation remains saved in the system for future use if desired. # Re-write the justification. The Submit button will send the form back to the manager for endorsement. # Click Submit as is. The form will go to the Board showing the manager's non-endorsement and reason. A manager can also be the author of a recommendation. The form is exactly the same, but is titled "Manager's Recommendation" and the roles are reversed. The form goes to the supervisor for endorsement. A manager can also re-write the justification that was submitted by the supervisor, and/or change the type of recommendation from promotion to award, or from award to promotion. If either is done, the recommendation automatically changes to a Manager's Recommendation, and it will go back to the supervisor for endorsement. The supervisor has the same options! A recommendation only goes to the Board if the justification and type have remained completely unchanged, the Endorsed or Not-endorsed option has been selected (with a reason provided for a non-endorsement), and the Submit button has been clicked. Manager's Recommendations endorsed by supervisors and Supervisor's Recommendations endorsed by managers are treated exactly the same way by the Board. Neither is better or worse than the other. Both are collaborative efforts. Special Achievement Anyone can recommend anyone for a Special Achievement Award. The form does not provide links to the employee's ARS report and résumé until it is being looked at by an authorized person such as the supervisor, the manager or members of the Board. Recommendations from persons other than the supervisor and manager go to both the supervisor and the manager for endorsement. Direct A manager, or supervisor, or anyone, may choose to send a recommendation to the Board without seeking an endorsement, though that is very rare and unlikely to be successful. To do this, the person submitting the recommendation must select the "Not endorsed" box for the supervisor and/or manager and must provide an explanation as to why the recommendation is being submitted without obtaining an endorsement. (The "Endorsed" option can be selected only by the named supervisor or manager, and will be digitally signed. The "Not endorsed" option can be selected by anyone and will also be digitally signed. A reason must always be stated whenever the "Not endorsed" option is selected.) Disciplinary A similar form exists for recommending disciplinary action. The title of the form is different, and the form is routed to a separate, Disciplinary Board. However, everything else is the same. Typically, a supervisor recommends and a manager endorses, but anyone can recommend, and anyone can submit the form with or without endorsements, though the same "Not endorsed" box(es) must be checked, and explanation(s) provided. Résumés For assignments and promotions, managers need to know more about the whole person, including specific types of experience and accomplishments. The employee résumé provides this information. An ARS résumé is a very concise outline of the employee's education and career. Résumés are generated automatically for all employees based on information provided by the Human Resources element. But it is up the employee and supervisor to provide additional detail by editing the résumé. Form The résumé is just a simple outline. Selecting an item in this outline causes a box to appear containing details about the item that was selected. If the person viewing the résumé is the employee or the supervisor, these details can be edited. Education, employment history, assignments, corporate course work and awards are listed automatically. Below the listing for each is a space for the employee (or supervisor) to enter a short statement of an associated achievement. When that space is filled in, a space for listing a next achievement appears, and so on. For each, the details box should contain an impact statement and any other information necessary to understand the achievement. The order of the list can be rearranged at any time. Validation A valid ARS résumé must be digitally signed by two persons, the employee and the supervisor. When the employee edits the résumé, the supervisor is notified and must review the résumé apply a digital signature. If the supervisor makes changes, the employee is notified and must do the same. This process continues until the exact same version is signed by both. All versions are retained, but only the validated versions, beginning with the most recent, are accessible to others. Earlier validated versions are highlighted in the list of all versions, which is called the "history" of the résumé. If changes are made by the employee to sections of the résumé that relate to an earlier assignment, the current supervisor can, and should, compare those changes with the version of the résumé that was validated by the earlier supervisor. Access Exactly as with ARS reports, the employee, the supervisor, the manager, and members of Boards can always view the résumé, but others must request access, and the employee is the one who decides. When seeking a new assignment, an employee should always grant access to the recruiting manager and supervisor. Frequently Asked Questions ; How many people can I rate as the Best?: Only one person can be rated as the Best for any one factor. If you select this option for another person, the person whom you earlier rated as the Best will automatically be re-rated as Superior. ; How many people can I rate Among the worst?: Only three persons can be rated as the Among the Worst for any one factor. If you select this option for an additional person, one of those whom you earlier rated as the Among the Worst will automatically be re-rated as Inferior. The system selects the one with the highest consensus rating for that factor. In the case of a tie, the system selects the one whose rating you submitted earliest. ; Must I rate someone as the Best or Among the worst?: No! You are only required to exercise good judgement. If the best or worst persons that you know are not among those that you are required to rate, then you should rate nobody as the Best and nobody as Among the Worst. ; Why is the option of No opinion sometimes not available?: It is not available only for persons on your Core List. No person should be on your Core List unless you and your supervisor agree that you are in a position to have a well-informed opinion on all of the five rating factors. Remove the person from your Core List if this is not true. ; Is there anything wrong with just selecting No opinion when available?: You should select no opinion if you are not in a position to have an independent, well-informed opinion about a factor as it pertains to that person. However, if you do have an opinion, you should express it. If No opinion is selected for three or more factors, the person will be removed from your Extended List and you will be required to rate another person. ; Should I view a person's résumé before rating the person?: No! Permission to view a résumé should be neither requested nor granted as part of the ratings procedure. Ratings are meant to be based on the your first-hand familiarity with the person being rated, and not on what you have read about the person in his/her résumé. ----